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ABSTRACT 

While the aptitude treatment interaction (ATI) 
approach to educational measurement emphasizes establishing salient 
learner characteristics, systematic formative evaluation provides 
ongoing evaluation for instructional program modification. Systematic 
formative evaluation appears more tenable than ATI for developing 
individualized instructional programs. This meta-analysis 
investigates the effects of systematic formative evaluation of 
educational programs on student achievement. Twenty-one controlled 
studies generated 95 relevant effect sizes, with an average effect 
size of .72. The magnitude of effect size was associated with 
publication type, data evaluation methods, and use of behavior 
modification. Findings indicate that unlike reported ATI approaches 
to individualization, systematic formative evaluation procedures 
reliably increase academic achievement. This suggests that, given an 
adequate measurement methodology, practitioners can inductively 
formulate successful individualized educational programs. 
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Runnino Head: Systematic Formative Evaluation 



Abstract 



This meta-analysis investigated the effects of formative evaluation procedures 
on 3tudent achievement. The data source was 21 controlled studies , which gen- 
erated 95 relevant effect sizes, with an average effect sizes of .72. The mag- 
nitude of the effect of formative evaluation was associated with publication 
type, data-evaluation method, and use of behavior modification. Implications 
for practice are discussed. 
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h Quantitative Synthesis ot Effects oi Formative Evaluation on achievement 



An essential purpose of educational measurement is to generate information 
with which to formulate instructional programs (Glaser & Nitko, 1971). Given a 
model of individualized instruction wherein students are taught not only at 
different rates but also by means of different instructional methodologies, 
measurement serves two major functions. First, it provides a description of 
the learner to guide the selection of an initial set of instructional proce- 
dures. Second, as the student engages in the instructional process, measure- 
ment generates information with which the effectiveness of the initial educa- 
tional program can be evaluated and modified as required. Theoretically, these 
two purposes appear to complement one another. Nevertheless, in practice, they 
have become associated with markedly different approaches to the development of 
individualized instructional programs. 

The first is an Aptitude-Treatment Interaction (ATI) approach. It exem- 
plifies the first measurement function, the initial description of learners. 
ATI proponents presume that specific learner characteristics, or aptitudes, in- 
teract predictably with certain types of instructional programs, or treatments, 
to produce comparatively strong student learning. Thus, with an ATI approach, 
the development or selection of educational programs is derived from a prior 
explication of learner characteristics. It is a deductive approach to formu- 
lating educational programs. 

In theory, the number of possible ATIs is limited only by our capacity to 
generate learner characteristics and related educational programs. M ot sur- 
prisingly, this perspective has inspired much research activity Into possible 
salient learner characteristics (see Snow & Lohraan, 1984) and models of in- 
struction (see Lloyd, 1984). Nevertheless, there are several important prob- 



ERIC 



4 



BEST 




Effects of Systematic 
2 



leras in basing educational programs on initial diagnoses of learner character- 
istics* First, at present, there is incomplete conceptualization of students 1 
cognitive abilities (Ysseldyke, 1979). Second, and relatedly, available tests 
of learner characteristics do not possess appropriate technical qualities 
(Salvia & Ysseldyke, 1981). Third, evidence indicates the manner in which 
these tests often are administered (e.g., in one sitting and by an unfamiliar 
examiner) may discriminate systematically against select groups of students 
(Fuchs & Fuchs, 1985; Fuchs, Fuchs, Power, & Dailey, in press). Fourth, knowl- 
edge concerning interactions among learner and teacher characteristics, educa- 
tional treatments, and classroom environments is far from complete (Ysseldyke, 
1979). These problems associated with an ATI approach appear serious* In all 
likelihood, they contribute to the fact that current research does not support 
the use of ATI approaches to improve achievement among special education stu- 
dents (see Lloyd, 1984) • 

The second, contrasting approach to developing individualized educational 
programs is systematic formative evaluation* Whereas an ATI approach empha- 
sizes the importance of the first purpose of educational measurement, estab- 
lishing salient learner characteristics, systematic formative evaluation em- 
bodies educational measurements second major function: ongoing evaluation and 
modification of proposed programs* Specifically, this approach employs regular 
monitoring of student performance under different instructional procedures* 
The purpose of this monitoring is to provide a data base with which individual- 
ized programs may be developed empirically. Thus, systematic formative evalua- 
tion is an inductive, rather than deductive, approach to developing instruc- 
tional programs. 

There are at least three reasons why, at: present, systematic formative 
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evaluation appears more tenable than ATI as a general strategy to develop Indi- 
vidualized instructional programs. First, its inductive nature avoids reliance 
on initial diagnoses of learner characteristics when there are incomplete con- 
ceptualizations of the relation between students 1 abilities and educational 
treatments. Second, its measurement procedures have been shown to be psycho- 
raetrically acceptable, whereas many ATI-related measures are seemingly inade- 
quate. Third, and relatedly, it requires repeated measurement by classroom 
teachers in familiar classroom settings, which appears more ecologically valid 
and less reactive than the use of traditional assessment procedures associated 
with typical ATI approaches. 

Moreover, systematic formative evaluation's repeated use of technically 
adequate measurement procedures appears consonant with public demand for ac- 
countability in the schools, as reflected in legislative action such as 
PL 94-142 (Deno & Mirkin, 1977). Nevertheless, there has been no attempt to 
Integrate available research on systematic formative evaluation or to quantify 
the magnitude of effect associated with such an approach to formulating indi- 
vidualized programs. This lack ol research contrasts sharply with the numerous 
integrations of research on ATI-related strategies in special education (e.g., 
Arter & Jenkins, 1977, 1979; Hammill & Larsen, 1974; Hammill & Wiederholt, 
1973; Kavale, 1981; Tarve : & Dawson, 1978). Consequently, the purpose of the 
current investigation was to conduct a meta-analysis of studies exploring the 
effects of systematic formative evaluation of educational programs on academic 
achievement* 

A previous meta-analysis investigated a related aspect of formative evalu- 
ation, corrective feedback (Lysakowskl & Walberg, 1982). However, the studies 
constituting that meta-analysis addressed only the effects of student feedback. 
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As Linn (1983) has noted, teacher feedback also is critical to the use of test- 
ing for formative purposes. Therefore, the present study contributes to the 
previous data base by quantifying the effects of formative feedback to teachers 
for the purpose of empirically developing individualized instructional pro- 
grams. 



Search Procedure 

The search for pertinent studies comprised four steps* First, employing 
the Thesaurus of Psychological Index Terms (APA, 1982), multiple descriptors 
ware generated from key topic-related terras. For example, student achievement 
alternately was Identified by "student progress," "goal attainment," and "edu- 
cational effects." Second, these terras facilitated a computer search of three 
on-line data bases: (a) ERIC, a data base of educational materials from the 
Educational Resources Information Center consisting of abstracts from Research 
in Education and Current Index to Journals in Education ; (b) Comprehensive 
Dissertation Abstracts ; and (c) Psychological Abstracts * Third, employing sim- 
ilar key descriptors, a manual search was conducted of five educational jour- 
nals for the years 1973 through 1983. These journals were: American Educa- 
tional Research Journal , Journal of Learning Disabilities , J ournal of Preci- 
sion Teachin g, Journal of Special Education , and Learnin g Disability Quarterly. 
Fourth, titles in the reference sections of investigations discovered by these 
efforts were explored for additional studies. 
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Criteria for Relevant Studies 

A study was considered for inclusion if it employed a control group to 
evaluate the effects of providing systematic formative evaluation to teachers 
concerning the academic performance of preschool, elementary, and/or secondary 
students. Studies were excluded that (a) monitored nonacademic behaviors, (b) 
primarily focused on the use of behavior raodif ication, while employing time 
series to test experimental effects, (c) provided test feedback only to stu- 
dents, and/or (d) employed college-age subjects. 

The search yielded 29 studies that met the inclusion criteria. From these 
studies, 8 were eliminated because of insufficient data tor calculating meta- 
analytic statistics. 

Data Extracted from Each Study 

Guidelines were established to ensure that each relevant effect was 
counted only once in analyses and that papers reporting results of the same 
study were grouped within analyses as one investigation. * 

Effect size* Results of the studies were transformed to a common metric, 
effect size, defined here as the difference between the treatment means, di- 
vided by the control group standard deviation. For purpose of analysis, an ef- 
fect was given a positive sign if subjects achieved greater scores in the sys- 
tematic formative evaluation treatment. For studies reporting relevant means 
and standard deviations for the systematic formative evaluation and control 
groups, effect sizes were calculated from these statistics. For studies not 
reporting means and standard deviations, effect sizes were calculated from 
other statistics, such as JF or £-values (see Glass, McGaw, & Smith, 1981). Be- 
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fore averaging effect sizes, each one was converted to an unbiased effect size 
(UES) to correct for the inconsistency in estimating true from observed affect 
sizes (Hedges, 1981 )• The difference between the observed and unbiased effect 
sizes was neglible (X * #019, SD - #025) as has been demonstrated elsewhere 
(Bangert-Drowns, Kulik, & Kulik, 1983). Nevertheless UESs were employed to in- 
sure the mathematical tractability of the data* 

Meta-analytic Z. Results from the 21 studies were combined to determine 
the unweighted Stouffer meta-analytic Z_ (Rosenthal, 1978). This statistic per- 
mits computation of the probability that the combined effect of children f s 
greater achievement scores in the systematic formative evaluation treatment 
would occur by chance. It was derived by changing the ^-values of all effects 
to scores, summing them, and dividing this sum by the square root of the num- 
ber of studies included. When calculating a score for studies in which mul- 
tiple dependent variables were analyzed, a median £-value was calculated for 
each study and its associated £ score was used in the meta-analysis (see Rosen- 
thal & Rubin, 1978). 

Methodological and Substantive Study Feature s 

Methodological study features. The effects of systematic formative evalu- 
ation of pupils 1 academic progress were related to three methodological vari- 
ables that were coded for each study. 

U Publication type . This refers to a description of the kind of litera- 
ture in which the studies were found. Coded values included "journals," "dis- 
sertations," and "nonpublished" studies such as ERIC reports, conference pres- 
entations, and solicited manuscripts. 

2. Publication year. This variable was coded "before 1975," "between 
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1975 and 1979, M and "between 1980 and 1984." 

3. Quality of Study . Each study was coded as ''poor/ 1 "fair/ 1 or "good. M 
To accomplish this, raters analyzed studies to Identify "serious" and "less 
serious" threats to Internal validity. "Serious" threats Included (a) unequiv- 
alent subject groups, (b) confounded experimental treatments, and (c) nonrandom 
assignment of subjects to treatments. Examples of "less serious" threats were 
(a) the use of technically inadequate dependent measures, (b) uncontrolled ex- 
aminer expectancy, (c) unchecked fidelity of treatment, (d) the employment of 
Inappropriate statistical unit of analysis, and (e) Inadequate teacher train- 
ing. "Poor" quality studies were identified on the basis of at least one seri- 
ous threat and/or because of a minimum of at least three less serious design 
flaws. Investigations were considered "fair" in quality if they were free of 
serious threats and evidenced no more than two less serious methodological 
problems. "Good" quality denoted studies displaying no more than one less ser- 
ious methodologial problem. 

Interrater agreement on each of the three methodological variables, based 
on two raters * evaluations of eight randomly selected studies (38% of the sam- 
ple), ranged from 75% to 100%. Average agreement across the three methodologi- 
cal features was 92%. 

Substantive study features . There were six substantive variables. 

1. Behavior modification . Studies incorporating behavior modification as 
part of a formative evaluation treatment were distinguished from those investi- 
gations that did not use this adjunct treatment. 

2. Data display . Investigations in which teachers were required to graph 
student performance data were differentiated from those in which teachers were 
asked simply to record data. 
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3. Data evaluation * Studies were identified on the basis of whether par- 
ticipants (a) were required to employ explicit, systematic data-evaluation 
rules that indicted when and/or how they were to introduce programmatic 
changes , or (b) were permitted to judge for themselves when and how to make 
changes in students' programs. 

4. Grade level * Subjects 1 average grade levels were aggregated into 
"preschool 3 through primary ," "intermediate, " or "junior and senior high" 
groups* 

5. Measurement frequency * Studies were noted for the frequency with 
which student performance was measured: 2 f 3, or 5 times per week* 

6. T reatment duration * Study length was coded in terras of "less than 3 
weeks," "3 to 10 weeks," or "greater than 10 weeks." 

Two raters independently coded the six substantive features in eight ran- 
domly selected studies (38% of the sample). Interrater agreement^ for the sub- 
stantive features ranged from 75% to 100%. Average agreement across all six 
substantive variables was 86%. 

Results 

Overall Effects 

Results of the 21 studies were combined to provide three interrelated 
aggregate descriptions of the effects of systematic formative evaluation: un- 
biased effect size (UES), percentage of distribution nonoverlap, and meta- 
analytic Zj 

The overall mean UES was .72 (SD « .88; SE - .09, jt (94) • 7.97, £ < .001. 
In terms of the percentage of nonoverlap between experimental and control group 
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distributions, U3 (Cohen, 1977), a UES of .72 indicates that the upper 50% of 
the experimental group distribution exceeds approximately 76% of the control 
group distribution. In terms of the standard normal curve and an achievement 
test scale with a -population mean of 100 and a standard deviation of 15, the 
integration of formative evaluation with instruction would raise the typical 
achievement outcome score from 100 to 110.80, or from the 50th to 76th percen- 
tile. 

The meta-analytic Z was 4.43, £ < .001, indicating that it is highly un- 
likely that the combined effect of students 1 greater achievement scores in the 
systematic formative evaluation treatment occurred by chance. Credence in a 
statistically reliable meta-analytic may be compromised by the suspicion that 
researchers do not report nonsignificant results (Greenwald, 1975). Rosenthal 
(1979) described a method for determining the number of unreported null effects 
that would be needed to reduce a meta-analytic £ to nonsignif icance. The 
larger this M f ail-safe N , ,# the more confidence one can have in the reliability 
of a meta-analytic result. This investigation's fail-safe N was 131, indicat- 
ing that it would take 131 studies summing to a null result to raise the prob- 
ability of the meta-analytic £ beyond .05. 

Relation Between UKSs and Study Features 

Methodological features . Table 1 displays data for the UESs by methodo- 
logical features of the effect sizes. There was one significant effect for 
type of publication. As indicated in Table 1, a follow-up Scheffe analysis in- 
dicated that the mean UES associated with reports published in journals was 
statistically significantly greater than the average UES associated with unpub- 
lished'' studies. 
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Substantive features . Table 1 also shows UESs by substantive features. 
There were two statistically significant effects. First, the data-evaluation 
variable yielded a significant £ value, with the mean UES greater for the use 
of data-evaluation rules than for teacher judgment. Second, the factor behav- 
ior modification resulted in a significant difference; the average UES was 
greater when behavior modification procedures were incorporated as part of the 
experimental treatment. 



The purpose of this meta-analysis was to determine the effects of system- 
atic formative evaluation of educational programs on academic achievement. Re- 
sults indicated the use of systematic formative evaluation procedures signifi- 
cantly increased students 1 school achievement, both statistically and pratical- 
ly# The mean effect size of .72 was reliably different from zero. It suggests 
that one can expect students whose programs are monitored systematically and 
developed formatively over time to achieve, on average, almost three-quarters 
of a standard deviation higher than students whose programs are not systematic- 
ally monitored and developed formatively. 

Moreover, this finding generally was robust over several methodological 
features associated with the effect sizes: Neither quality of study nor publi- 
cation year appeared to mediate or moderate formative evaluation effects* Only 
publication type yielded a statistically significant difference, wherein effect 
si2es associated with studies published in journals were higher than those de~ 
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rived from unpublished manuscripts* Such a finding might be anticipated given 
the tendency of journals to reject studies that fail to yield reliable results 
(Rosenthal, 1978), and given the related suspicion that researchers do not re- 
port nonsignificant results (Greenwald, 1975) # Nevertheless, the meta-analytic 
£ analysis indicated that it would require the addition to this meta-analysis 



of as many as 131 studies summing to a null result to reduce findings to non- 
significance* 

Findings were robust across not only methodological features, but also 
substantive variables* Specifically, systematic formative evaluation was simi- 
larly effective regardless of students 1 age, treatment duration, the frequency 
with which measurements were taken, or whether student data were graphed* Non- 
significant findings associated with some of these variables may be explained 
at least partially by the low number of effect sizes for certain coded vari- 
ables* For example, the mean effect size associated with the practice of 
graphing data *as 3*5 times greater than the average effect size associated 
with simply recording such information; however, there were only seven effect 
sizes for recording* 

Despite the general robustness of findings for substantive variables, two 
substantive study features produced reliably different effect sizes, and appear 
to reflect critical dimensions of effective formative evaluation systems* 
Specifically, effect sizes connected with the use of behavior modification in 
addition to systematic formative evaluation were reliably higher than those 
representative of systematic formative evaluation only* This finding is conso- 
nant with previous research* For example, Bloom (1984) reported an effect size 
of 1*20 on student achievement for reinforcing students 1 academic behavior* 
Thus, it Is not surprising that incorporating reinforcement as part of system- 
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atic monitoring procedures would produce differentially greater student 
achievement* 

A less predictable finding of the current study was the significant dif- 
ference associated with data-evaluation methods* When teachers were required 
to employ data-utilization rules, effect sizes were higher than when data were 
evaluated by teacher judgment. Data-evaluation rules required practitioners to 
analyze student performance at regular intervals and, If the data suggested 
certain patterns, to Introduce Instructional changes Into a student f s educa- 
tional program* For example, Fuchs, Deno, and Mlrkln (1984) required teachers 
to calculate a line of best fit through every 7 to 10 data points* If a line 
of best fit was less steep than the goal line, running from baseline to the In- 
tersection of the criterion performance and the goal date, teachers were re- 
quired to Institute a programmatic change* Results suggest that, In order to 
effect greater learning for pupils, teachers might employ explicit, systematic 
rules to evaluate the data they collect* This finding Is In concert with pre- 
vious work (Baldwin, 1976; Tindal, Fuchs, Christenson, Mlrkln, & Deno, 1981; 
White, 1974), demonstrating that although teachers may collect student perform- 
ance data according to designated time schedules, they frequently do not employ 
those data meaningfully to modify students 1 educational programs* 

Therefore, findings of the current study Indicate tha^ r he use of system- 
atic formative evaluation procedures reliably increases academic achievement, 
and that effects may be enhanced when teachers also employ behavior modifica- 
tion and data-evaluation rules* The apparent effectiveness of systematic form- 
ative evaluation suggests that, given an adequate measurement methodology, 
practitioners can Inductively formulate successful Individualized educational 
programs* This conclusion contrasts with a body of literature Indicating that 
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ATI approaches to individualization, wherein different instructional programs 
are deductively formulated from explications of learner characteristics, fail 
to enhance achievement* The use of systematic formative evaluation and result 
ing development of effective individualized programs might be considered by 
those who, In their astute criticisms of ATI approaches, also have questioned 
the validity of individualized instruction (see, for example, Lloyd [ 1 984 ] )• 
Given results of this meta-analysis, we believe such questioning of the legiti 
macy of individualized instruction may represent a case of "throwing the baby 
out with the bath." 

Current findings must be considered in light of at least two possible 
methodological problems. First, a limited number of researchers dominate the 
group experimental literature in systematic formative evaluation. Specifical- 
ly, one team of investigators (Deno et al« ) accounted for eight reports em- 
ployed in the meta-analysis and one researcher (Beck) accounted for four 
studies. While such a pattern may be problematic because it may inflate meta- 
analynic findings (see Slavin, 1984), post-hoc analysis indicates that the ef- 
fect sizes associated with these sets of researchers did not inflate results, 
but rather tended to underestimate the average effect size. A second conceri 
in this quantitative synthesis is that a relatively small number of investiga- 
tions produced a large number of effect sizes, a situation that results in do- 
pendency among effect sizes. This methodological problem commonly is associ- 
ated with meta-analytic research. It warrants additional attention by those 
concerned with the development of meta-analytic methodology, and current finc- 
tngs must be understood within the confines of this limitation. 
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Footnotes 



1 One paper authored by Haring (1971) and two additional reports by Haring 
and Krug (1975a, 1975b) described aspects of the same investigation. Only non- 
redundant effect sizes were extracted from these reports and f when analyses re- 
quired that effect sizes be grouped by investigation, such as the meta-analytic 
Z_, these effect sizes were grouped as one investigation. Therefore, although 
it is reported that 21 studies were employed in the meta-analysis, 23 appear in 
the appendix due to the separate listings of the Haring and Haring and Krug 



2 Interrater agreement was calculated using the following formula (Coulter 
in Thompson, White, & Morgan, 1982): Percentage agreement 3 agreements between 
rater A & rater B / (agreements between A & B + disagreements between A & B 
+ omissions by A + omissions by B)« 

Only one study included subjects whose average age was at the preschool 
level; thus, effect sizes for preschool children wer* grouped with those asso- 
ciated with primary grade students* 



papers* 
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Table 1 



Means, Standard Deviations, £ values, and Significant Scheffe Contrasts 
on UESs by Methodological and Substantive Study Features 
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.80 


47 


Quality of Study 








A, Good 


.85 


.45 


19 


B, Fair 


.71 


.99 


66 


C. Poor 


.54 


.63 


11 


Substant Ive 








Behavior Modification 








A, With behavior modification 


1.15 


1.31 


30 


B. Without behavior modification 


.52 


.48 


66 


(Table 


1 continued 


on p. 


20.) 



22 



.67 



.43 



2,93 



2,93 



11.96 b 1,94 



A > B 



BEST COPY 



7 



n 
rr 



CO 

^< 

05 

rr 
fO 



rr 

O 

23 



Table 1 (Continued) 
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m m-m-mm mm mm -m-m mm 

Mean 
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SD 
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Data Display 
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A. Graphed 
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B. Recorded 
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7d 




Data b valuation 








9. lie 1,94 
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.95 


1.08 
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Measurement Frequency 
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W • I * CM J* Am J 


77 

. 9 I 


* J u 


O J 




Treatment Duration 








.88 2,93 


A. Fewer than 3 weeks 


.60 
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a N_ represents number of UESs not number of studies. 
b £ < .001. 

C £ < .01. 

^ The small _N in these categories may result in unstable estimates of UES. 
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